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MEANS AND METHODS OF SOUND SYNTHESIZING 



FIELD OF THE INVENTION 

This invention relates to means and methods of sound synthesizing and, 
5 more particularly to means and methods of synthesizing the sound of musical 
instruments and the like. More specifically, although of course not solely limited 
thereto, this invention relates to the synthesizing of the sound of a string 
instrument such as violin, bass, cello and piano. 

BACKGROUND OF THE INVENTION 

10 Means and methods of sound synthesizing are widely used in the 

production or reproduction 'of sound by electronic means. For example, music 
(including vocal and instrumental), tones (for example, ringing tones and other 
tones used in mobile telephones) and background sound are commonly 
synthesized by a digital signal processing means. 

15 It is well known that an audio signal can be represented by a Fourier series 

and decomposed into a series comprising weighted harmonics of the fundamental 
frequency. For example, Jean-Claude Risset and Max v. Mathews in the article 
entitled "Analysis of Musical Instrument Tones" Physics Today, vol. 22, no.2, pp. ■/ 
23-30 (1969) advanced that the temporal evolution or the evolution in time of the 

20 spectral components of a sound is of critical importance in the determination of 
timbre. They suggested that to simulate a natural sound, the amplitude of each 
harmonic should be individually controlled as a function of time. In addition, at 
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least for the production of the trumpet tones, that the attack envelope (that is, the 
initial envelope characteristic for the trumpet tone) has a distinctive characteristic 
. in that during the attack and also decay portion of the sound, the energy of the 
various frequency components evolve in complicated ways. To implement the 
Risset theory with known techniques requires a complex digital computer which 
individually simulates each frequency component. As a result, a real time music 
synthesizer of the digital type has not been commercially feasible. 

While the Fourier series provides an accurate representation of the signal 
or sound to be reproduced, the processing power overhead, for example, for 
calculating up to 20 or 30 harmonics of a particular musical instrument, may- be 
prohibitively high. Such a high processing power may not be available in portable 
devices not solely dedicated for music or sound generation, for example, mobile 
phone with tone or music, generator. Hence, it is beneficial if there can be 
provided means and methods for synthesizing sound of a reasonable quality 
without requiring a high processing power. Furthermore, the increasing popularity 
of polyphonic ring tone or music synthesis in mobile phones have made the 
benefits of a more efficient sound synthesizer even more apparent. The FM 
synthesizing method proposed in US 4,018,121 by Chowning is a well known type 
of sound synthesizing method without requiring a very high processing power. 

However, it is recognized that the sound or timbral quality of the musical 
sound synthesized by the FM synthesizing method or other more efficient sound 
synthesizing methods are not satisfactory. In particular, the synthesized sounds of 
string instruments such as violin, cello, guitar or piano lack the "richness" to be 
real enough to be appreciated. Hence, it would be highly beneficial if there can be 
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provided improved means and methods of sound synthesizing so that musical or 
other sounds can be generated with a reasonable degree of genuineness without 
requiring excessive computational overhead. 

OBJECT OF THE INVENTION 

Hence, it is an object of the present invention to provide means and 
• methods of sound synthesizing, especially musical sound, synthesizing with a 
reasonable degree of genuineness while without requiring an excessive 
computational overhead. At a minimum, it is an object of this invention to provide 
means and methods of sound synthesizing for the benefit and choice of the public. 

SUMMARY OF THE INVENTION 

Broadly speaking, the present invention has described a method of 
synthesizing the sound of a musical instrument, including the steps of:- 

• obtaining samples of the sound of said instrument, 

• analysing the harmonics of said samples of said sound, 

• selecting harmonics of said sampled sound according to prescribed 
characteristics of the envelop of said harmonics for synthesizing 
harmonics of the synthesized sound, 

• grouping harmonics of said sampled sound of similar envelop 
characteristics and obtaining temporal characteristics of the group of 
harmonics from constituting harmonics of the same group, 
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• synthesizing a plurality of synthesized harmonics of the synthesized 
sound, wherein at least some of the synthesized harmonics are 
synthesised from one of the envelops of the harmonics of a group and 
conditioned by the temporal characteristics of the constituting harmonics 
of that group. 

Preferably, said prescribed characteristics for selecting a harmonic including 
selecting a harmonic with more salient variation in amplitude over-time. 

Preferably, a plurality of selected harmonics of said sampled sound being 
group added to form a synthesized harmonic of the synthesized sound. 

Preferably, said synthesized .harmonic obtained by group addition being 
scaled up or down for generating other harmonics of said synthesized sound. 

Preferably, said synthesized sound being synthesized from a plurality of 
characteristic harmonics, a plurality of said characteristic harmonics having a 
substantially similar envelope. 

Preferably, the number of said plurality of characteristic harmonics does not 
exceed 4. 

Preferably, at least one of said characteristic harmonics being synthesized 
from a plurality of harmonics of said samples of said sound. 

BRIEF DESCRIPTION OF THE DRAWINGS 
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Preferred embodiments of the present invention will be explained in further 
detail below by way of examples and with reference to the accompanying 
drawings, in which:- 

Fig. 1 is a three-dimensional diagram showing the amplitude-time- 
5 harmonics relationship of a sample of a "Synstring" sound obtained by FM 
synthesis, 

Fig. 2 is a chart showing the amplitude-time relationship of the first 
harmonic of the Synstring signal of Fig. 1 , 

Figs. 3-6 show respectively the amplitude-time relationship of the second, 
10 third and fourth harmonics of the Synstring signal of Fig. 1 , 

Fig. 7 shows the amplitude-time chart of the fifth to the eighth harmonics of 
the Synstring signal of Fig. 1, 

Fig. 8 shows the wavetable of the first synthesized harmonic, 

Figs. 9 and 10 respectively show the wavetable for the second and the 
15 third synthesized harmonics; 

Fig. 11 shows the wavetable of the fourth group of the synthesized 
harmonics, 

Fig. 12 shows the synthesized harmonics of the synthesized sound from 
the four groups of synthesized harmonics and their respective wavetables, 

20 Fig. 13 shows, from top left and clockwisely, respectively, the envelops 

(amplitude-time) variation of the first to the fourth synthesized harmonic groups, 
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Fig. 14 shows the amplitude-harmonics and time chart of the four most 
dominant harmonics of the sampled sound, 

Fig. 15 is an amplitude-time diagram showing the waveform of the sound 
of a Synstring signal synthesized by the group additive synthesis of this invention, 

. 5 Fig. 16 shows the spectral diagram of the harmonics of the synthesized 

sound formed by the four groups of the synthesized harmonics. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

In order to obviate the need of a dedicated processor with a high 
processing power to provide synthesized sound or musical sound of a reasonable 

10 timbral quality, the present invention provides means and methods to synthesize a 
musical sound from a plurality of synthesized characteristic harmonics. The 
synthesized characteristic harmonics are then used to synthesize additional 
harmonics to provide the desirable timbral quality. Hence, the musical sound of a 
reasonable quality can be constructed from a selected plurality of synthesized 

15 harmonics, thereby alleviating the need of exhaustive competition. 

Referring firstly to the exemplary spectrum of Fig. 1 showing the 
harmonics of a Synstring signal sample of Fig. 1 and the schematic representation 
of 8 showing the more salient or dominant harmonics of the Synstring signal. It 
will be noted that among the 20 harmonics shown in the Figures, the amplitude- 
20 time variation of the first eight harmonics are particularly noticeable or salient. 
Hence, they are of substantial practical audio importance since their amplitudes 
are of those sufficient significance. Thus, it will be appreciated that the Synstring 
signal can be adequately represented by the first 8 harmonics. However, to 

IP\HALE\ \0222£904.doa fl 



synthesize the sound using all the 8 harmonics still requires a very substantial 
processing power as well as data storage which are not sufficiently efficient for 
many practical applications. In this context, it will be appreciated that the term 
"Synstring" is known to persons skilled in the art as the synthesized sound of a 
5 string instrument such as violin, viola, cello, guitar and piano, which has been 
used here as a convenient illustrative example. 

In this preferred embodiment, the plurality of the characteristic synthesized 
harmonics are synthesized from the 8 more dominant or salient harmonics of the 
sampled sound to produce a lesser plurality of synthesized harmonics for 
10 constructing the synthesized sound with the necessary timbral quality. 

In this example, 4 characteristic .synthesized harmonic groups are 
produced first. 

As can be seen from Figs. 1 and 7, the first harmonic of the sampled 
sound has the most dominant amplitude characteristics. Hence, it is selected as 
15 the first group of the synthesized harmonic. Thus, the envelope of the first 
harmonic of the sampled sound is also the envelope of the first synthesized 
harmonic of the synthesized sound. 

The second synthesized harmonic group is built from the second and the 
third harmonics of the sampled sound. Such a grouping selection is made 
20 because the second and the third harmonics exhibit similar characteristics on 
variations of amplitude with time (as reflected by the trend of the tooth-shaped 
envelope). 
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The envelope of the third harmonic of the sampled sound is used as a 
reference for synthesizing the envelope of the second group of synthesized 
harmonics of the synthesized sound because the envelope of the third harmonic 
has the larger and therefore more dominant amplitude in this group. 

5 The third group of the synthesized harmonic of the synthesized sound 

consists of the fourth harmonic of the sampled sound since it can be seen that the 
fourth harmonic has a unique amplitude-time variation which is very different from 
the remaining of the harmonics of the sampled sound. Hence, the envelope of the 
third group of the synthesized harmonics is also the envelope of the fourth 
10 harmonic of the sampled sound. 

As the remaining harmonics of the sampled sound, namely, the fourth to 
eighth harmonics, have a similar saw-tooth envelop trend and have comparable 
relative amplitudes, it is believed that they represent the similar timbral quality and 
could be grouped together to form one of the synthesized harmonics. 
15 Furthermore, as the shape of the saw-tooth envelop of the sixth harmonic is more 
salient, that is, the amplitude between adjacent peaks and troughs are more 
significant, the envelop of the sixth harmonic is chosen as a normalizing reference 
and envelop to be explained below. 

To construct the synthesized harmonics; a plurality of wavetables is 
20 constructed. As a convenient example, the wavetable has a table size of 128 
entries. As a full signal cycle can be represented by a cycle of 360°, each entry in 
the wavetable represents 2.8125°. For an exemplary 16 bit system, the signal 
amplitude can be resolved into 32767 levels. An exemplary wavetable 
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constructed for the first harmonic of the sampled sound is shown in Table A below 
as a convenient example. 



Degree 


Radian 


First harmonic 


32767 


0 


0 


0 


0 


! 2.8125 


0.049087 


0.049068 


1607 


5.625 


0.098175 


0.098017 


3211 










360 


6.2832 


0 


0 



TABLE A 



As the second synthesized harmonic group is constructed from the second 
5 and third harmonics of the sampled sound, the wavetable is constructed by adding 
the corresponding temporal amplitude values of the second and third harmonics. 
The amplitude values of the second harmonic set out in Table B is normalized by 
the scale of 0.988. This normalizing factor is obtained by dividing the value of the 
plague amplitude of the second harmonic which is 4359 on the 32767 level by the 

10 peak amplitude value 4413 of the third harmonic, that is, 4359/4413 = 0.988. The 
maximum sum obtained by adding 0.988 x amplitude of the second harmonic + 
the amplitude of the third harmonic is about 1.9 (referred to the entry under 
30.9375°). Hence, this sum is scaled down or normalized by the factor of 1.9 to 
ensure that the maximum does not exceed 1 or 32767. Then the same factor 1.9 

15 is used to scale up the time-amplitude envelope of third harmonic to compensate 
for the scaled down of the wavetable.The wavetable of the third group as 
constructed from the scaled down sum of Table B is graphically represented in 
Fig. 9 as the wavetable. 

As the third synthesized harmonic group consists only of the fourth 
20 harmonic of the sampled sound, the wavetable of the third synthesized harmonic 
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group is equal to the wavetable of the fourth harmonic of the sampled sound, as 
illustrated in Fig. 11. Next, the fourth synthesized harmonic group is further 
synthesized by group additive synthesis of the fifth to the twelve harmonics by 
adding corresponding temporal values of the amplitudes of the respective 
5 harmonics, as shown in more detail by the tabulation of Fig. C which shows a 
portion of the calculation. In constructing this fourth group of synthesized 
harmonic, the envelope of the sixth harmonic is used as a reference because of 
its more salient saw-tooth features. Before group adding the harmonics, the fifth 
harmonic is scaled up or normalized by the factor of 1.25 which is the ratio 

10 between the peak amplitude of the fifth harmonic (3389) and the sixth harmonic 
(2719) (1.25 = 3389/2719). This normalization before adding is repeated for the 
other harmonics. Although all the fifth to twelve harmonics are used in this 
tabulation, it will be appreciated that the harmonics beyond the eighth are already 
less significant and their inclusion is merely for further enhancement of a timbral 

15 quality while the inclusion of the fifth to the eighth harmonics would have given a 
reasonable timbral quality already. As can be seen from the "sum" column of 
Table C, the maximum value of the added amplitude is about 2.16. Hence, the 
added sum will be scaled down by factor of 2.2 to avoid overflow of the 16 bit 
system. Again the time-amplitude envelope of the sixth harmonic has to be scaled 

20 up by the same factor 2.2 to compensate for the scale down of the wavetable. 
After the four synthesized groups have been constructed, the four characteristic t ; 
synthesized harmonic groups will be used to construct the synthesized sound. 
Specifically, the first harmonic of the synthesized sound is constructed by 
multiplying the first wavetable with the first harmonic of the sampled sound. The 

25 second synthesized harmonic constructed by multiplying the second wavetable 
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(obtained by combination of the second and the third harmonics) with the scale up 
third harmonic of the sampled sound. The third synthesized harmonic is obtained 
by multiplying the third wavetable, which is obtained from the fourth harmonic of 
the original sampled sound, with the fourth scale up harmonic of the sampled 
5 sound. 

Next, the fourth synthesized harmonic group is constructed from the fourth 
wavetable and the sixth harmonic by multiplication since the sixth harmonic has a 
more dominant saw-tooth envelop. The amplitude-time envelope of the first to the 
fourth synthesized harmonic groups and the amplitude-harmonic-time spectral 
10 diagram are respectively set out in Figs. 13 and 14. 

The amplitude-time .envelopes of the first to the fourth synthesized harmonic 
groups are sliced into a plurality of intervals of 0.02 s width. Arrays containing the 
values of individual time-amplitude envelopes are constructed from the selected 
envelopes normalized by the scale up factor (i.e. envelopes of the first harmonic, 

15 third harmonic with scale up factor 1.9, fourth harmonic and sixth harmonic with 

i 

scale up factor 2.2). Due to the changes of the relative amplitude of the envelopes 
against each other, the temporal evolution characteristic of musical sound can be 
synthesized. Furthermore, the amplitude value of a particular synthesized 
harmonic group at a particular time is looked up from the array. The value, is then 
20 multiplied by the corresponding level value from the wavetable at the desired | ; 
frequency. Put simply, the wavetable is to synthesize the spectrum of a musical 
sound. The time-amplitude envelope is to synthesize the temporal evolution of a 
musical sound. These two are the most two important characteristics of 
synthesizing a musical sound. 
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As the respective wavetables for the particular synthesised harmonic contain the 
values of the relevant temporal evolution information of the respective constituting 
harmonics of the sampled sound, the multiplication by the wavetables of the 
selected harmonic envelope imparts the quality of the constituting sampled 

5 harmonics to the selected envelope. For example, the second synthesized 
.harmonic group is built on the time-amplitude envelope of the third harmonic of 
the sampled, by multiplying the envelope of the sampled third harmonic with the 
second wavetable which contains temporal evolution elements of both the second 
and the third sampled harmonics, the . temporal characteristics of the second and 

10 third sampled harmonics are imparted onto the second synthesized harmonic. 
This applies mutatis mutandis to the fourth synthesised group. 

As a specific example, the synthesizing of a 440 Hz Synstring signal at a sampling 
rate of 44 kHz is illustrated, As the lookup wavetable has a total of 128 entries, 
each entry on the wavetable will represent 344.5 Hz (44. 1 kHz/1 28). 

15 At 344.5Hz, the lookup address of the wavetable needs to be incremented 

by one to obtain the desired first harmonic wavetable. If we want to have a 
frequency of 440Hz is desired, the index will be exceeding 1 for the lookup 
address of the wavetable which is given by the following formula: 

e.g., the index will be 440Hz/344.5Hz= 1.277 

20 That means at each sampling at 44.1kHz, the lookup address will be 

incremented by 1.277 instead of 1 and the desired 440Hz can be represented by 
the 128 entries of the first harmonic wavetable. 
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To regenerate the sound of a Synstring signal at frequency 440Hz, the 
corresponding table and amplitude will be multiplied and the four groups of 
synthesized harmonics will be lumped together. 

An example of a simple program to synthesize the 440Hz Synstring sound 
5 is set out below. 

Accumulator : wavetable lookup address 
Index: increment at the desired frequency 
Table: table array of the wavetable above 

Coefficient: a calculated result from table array of the amplitude above, 
10 Synstring: the pcm output value 

At 44.1 kHz sampling, the output is calculated at the sampling as follows: 

Accumulator = Accumulator+Index; 
if (Accumulator^ 1 28) 

1 5 Accumulator = Accumulator-1 28; 

Synstring=Table1 [Accumulator]*Coefficient1 ; 

Synstring=Synstring+Table2[Accumulator]*Coefficient2; 

Synstring=Synstring+Table3[Accumulator]*Coefficient3; 

Synstring=Synstring+Table4[Accumulator]*Coefficient4; 

20 The coefficient is calculated at every 0.02sec as follows: 

Coefficient: the calculated result from amplitude and volume 

Scale: the scaling factor to normalize the volume of a musical instrument with 

other musical instrument 

Volume: the sound volume of the desired musical instrument 

25 

The amplitude loop-up from the amplitude envelope using the elapsed time 
as the lookup address will be as follows: 

For example, the elapsed time from the turn on of Synstring instrument at 
440Hz is 0.03sec, the amplitude at 0.02sec is used. If the volume is 10 and the 
30 scale factor is 5, the coefficient for amplitudel is 8852*10*5 = 442600. Hence, 
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442600 will be the Coefficients above until the end of 0.04 sec which will be used 
as the next value to 8852 in the line. 

The synthesized waveforms of the exemplary 440Hz Synstring sound in a 
spectral harmonics representation are shown respectively in Figs. 15 and 16. 

5 It will be noted from the spectral diagram of Fig. 16 that the spectrum of 

the synthesized sound obtained from the above group additive synthesis is more 
uniform than that obtained from FM synthesis or the original sample. It will be 
appreciated that by using group additive synthesis, less processing power and 
memory can be used to synthesize musical sound without noticeable difference 
10 compared to the actual or real instrument to the audience. In mathematical form, 
the synthesizing of the signal can be represented by the formula below. 

In general, audio signals can be represented by as follows: 

Where, 

S(t): Signal at time t, 
15 Ai(t): Amplitude of ith harmonic at time t, 
S(t)=Si Ai(t)*sin(2*7rWt), and 

i from 0 to n 

In the sound of Synstring, this is reduced to as follows: 

S(t) = B1(t)*sin(2*7t*f*t) + B2(t)*(0.988*sin(2*7c*2*f*t)+sin(2*Tc*3*rt))+ 

20 B3(t)*sin(2*7i;*4*f*t)+ 

B4(t)*(-1 .25*sin(2*7i*5*f*t))+sin(2*7t*6*f*t))+. . .) 

Where B(t) is the normalized amplitude envelope. 
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This allows the sound to be generated by only 4 table lookups, 4 
multiplications and 4 additions at the prescribed sampling rate which greatly 
reduces the processing power required. 

While the present invention has been explained by reference to the 
5 examples or preferred embodiments described above, it will be appreciated that 
those are examples to assist understanding of the present invention and are not 
meant to be restrictive. The scope of this invention should be determined and/or 
inferred from the preferred embodiments described above and with reference to 
the Figures where appropriate or when the context requires. In particular, 
10 variations or modifications which are obvious or trivial to persons skilled in the art, 
as well as improvements made thereon, should be considered as falling within the 
scope and boundary of the present invention. 

Furthermore, while the present invention has been explained by reference 
to the specific ground additive synthesis outlined above, it should be appreciated 
15 that the invention can apply, whether with or without modification, to other 
synthesizing scheme utilizing a plurality of the harmonics of the sampled sound to 
construct the synthesized sound without loss of generality. 
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CLAIMS 



1 . A method of synthesizing the sound of a musical instrument, including the 
steps of:- 

• obtaining samples of the sound of said instrument, 

• analysing the harmonics of said samples of said sound, 

• selecting harmonics of said sampled sound according to prescribed 
characteristics of the envelop of said harmonics for synthesizing 
harmonics of the synthesized sound, 

• grouping harmonics of said sampled sound of similar envelop 
characteristics and obtaining temporal characteristics of the group of 
harmonics from constituting harmonics of the same group, 

• synthesizing a plurality of synthesized harmonics of the synthesized 
sound, wherein at least some of the synthesized harmonics are 
synthesised from one of the envelops of the harmonics of a group and 
conditioned by the temporal characteristics of the constituting harmonics 
of that group. 

2. A method of claim 1 , wherein said prescribed characteristics for selecting a 
harmonic including selecting a harmonic with more salient variation in 
amplitude over-time. 
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3. A method of claim 1, wherein a plurality of sejected harmonics of said 
sampled sound being group added to form a synthesized harmonic of the 
synthesized sound. 

4. A method of claim 3, wherein said synthesized harmonic obtained by group 
5 addition being scaled up or down for generating other harmonics of said 

synthesized sound. 

5. A method of daim 1, wherein said synthesized sound being synthesized 
from a plurality of characteristic harmonics, a plurality of said characteristic 
harmonics having a substantially similar envelope. 

10 6. A method of claim 5, wherein the number of said plurality of characteristic 
harmonics does not exceed 4. 

7. A method of claim 5, wherein at legst one of said characteristic harmonics 
being synthesized from a plurality of harmonics of said samples of said 
sound. 
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